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Abstract 

This paper investigates the use of a cloud computing environment to facilitate the teaching of web 
development at a university in the Southwestern United States. A between-subjects study of students 
in a web development course was conducted to assess the merits of a cloud computing environment 
instead of personal computers for developing websites. The goal of using cloud being to ensure that 
each student had access to the same high-quality learning experience. The study also sought to 
determine the extent to which cloud computing could ensure efficient use of students' time through 
eliminating hardware and software troubleshooting. Finally, the study sought to assess the extent to 
which the use of cloud computing would enhance students' learning experience. 
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1. INTRODUCTION 

The web development course considered in this 
study is taught in-person at the junior level and 
exists as a required course for students within the 
IS department in a business college. The college 
has a highly structured curriculum with a strong 
common core. As a result, there is a reliance on 
this course to meet specific learning objectives to 
prepare students for future courses. The first half 
of the course covers the fundamentals of HTML, 
CSS, and Javascript. The second half of the 
course requires students to install a complex 
Integrated Development Environment (IDE) and 
then build an e-business website that features a 
range of HTML and CSS components along with 


user interaction mechanisms including forms and 
a database. 

Currently, instructors have students install web 
development software on personal computers 
and implement their projects in a standalone 
environment. Collectively, faculty have observed 
that student experiences with installing, 
configuring, and maintaining a web development 
platform on personal computers negatively 
impacts academic performance. One important 
drawback to this approach is that some students 
struggle to make the software operate on their 
computers. Time devoted to the installation is 
wasted as it detracts from learning objectives. 
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Faculty involved with the web course believe that 
cloud computing environments, which offer 
ubiquitous access to development systems, would 
improve student learning outcomes and 
satisfaction. Thus, the purpose of this study was 
to measure the potential difference in learning 
experience in a web development course between 
students with and without access to cloud-based 
services. 

2. LITERATURE REVIEW 

Web 

Web design is an important topic in computer 
science and information systems academic 
programs and students gain significant leverage 
in the job market with related skills (Ellis, 2007, 
Sridharan 2004). There is a broad consensus in 
the literature on the difficulty of teaching web 
development. Deshpande & Hansen (2001) 
referred to web development as a discipline 
among disciplines as it draws together skills from 
areas such as business, computer science, and 
art. 

Zhang & Dang (2015) suggest that difficulties 
with teaching web development result from 
needing to possess many skills beyond traditional 
programming using languages such as Java & C#. 
Web programmers also need skills which include 
server controls, data validation, site navigation, 
session validation, database, authentication and 
more. Del Fabro, de Almeida & Sluzarski (2012) 
offered three reasons for the difficulty of teaching 
web development which is 1) course content, 2) 
required infrastructure and 3) the environment of 
the university. The first two items reflect on the 
broad set of skills required by students and the 
required infrastructure to provide complex 
development environments for students' work. 
The third item argues that school often consists 
of many relatively trivial assignments that do not 
reflect complexities and dependencies students 
would face in real-world projects. 

Connolly (2011) offered a review of computer 
science education literature. In the review, 
Connolly revealed that the most common 
approach to teaching web development in North 
America is to fit all needed material into a single 
upper-level course. The UK differs in that 
programs offer web develop as stand-alone 
degrees with topics divided into separate and 
distinct courses. 

The literature regarding web development is 
consistent in demonstrating that there is much 
content to be taught/learned and that this 
content may be too much for a single course. The 


literature, particularly Zhang & Dang (2015) and 
Fabro, de Almeida & Sluzarski (2012), is also 
consistent in calling for a learning environment 
that includes not only web programming but also 
the related duties of a web developer including 
controls, database integration, authentication, 
data/session validation and much more. 

Cloud 

While cloud computing appears to be a relatively 
new concept, the technology was first introduced 
in the 1960s (Marston et al., 2011). The concept 
has surfaced in such forms as Application 
Management Services (AMS) and Application 
Service Providers. There is disagreement on the 
definition of cloud computing due to factors such 
as the approach taken (Prelas Kovacevic, 
Spoljaric, & Hegyi, 2012). In this paper we adopt 
the definition offered by the National Institute of 
Standards and Technology (NIST) which states: 

"Cloud computing is typically presented as a 
service model for enabling ubiquitous, 
convenient, on-demand network access to a 
shared pool of configurable computing resources 
(e.g., networks, servers, storage, applications, 
and services) that can be rapidly provisioned and 
released with minimal management effort or 
service provider interaction." 

NIST's five essential characteristics of cloud 
computing are useful: on-demand self-service, 
broad network access, resource pooling, rapid 
elasticity and measured service. The technology 
and cost improvements represented by cloud 
computing have led to the technology being called 
a "genuine information technology revolution" 
(Morrison, 2011). Conn & Reichgelt (2013) 
referred to cloud computing as the "fifth major 
paradigm shift in computing". IT experts predict 
that cloud computing will be the "dominant IT 
service delivery model" by the end of the decade 
(Jararweh et al., 2014). 

Universities recognize the value of cloud 
computing (Mircea & Andreescu, 2011). The 
technological concept is ranked as one of the top 
priorities for higher education (Lowendahl, 2012). 
Sultan (2010) focused on the power of cloud 
computing to better utilize resources and reduce 
risk for academic institutions. 

Ercan (2010) addressed not only the ability of 
cloud computing to offer cost savings to support 
university operations, but also benefits for 
students and academic programs. There are 
examples of cloud computing being adopted for 
specific courses or objectives such as Lawler and 
Joseph (2012) who used cloud computing in an 
entrepreneurship program and Chen et al. (2012) 
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who integrated cloud computing into a set of IS 
and CS courses. 

Grossniklaus and Maier (2012) taught on the use 
of NoSQL and VoltDB in the cloud, and Mrdali 
(2011) offered an alternative Business 
Intelligence course using cloud technology to 
create a dynamic and cost-effective learning 
environment. Conversely, Lawler (2011) defined 
a more comprehensive and structured program 
that leverages cloud technology to provide an 
education platform for teaching CS and IS. 

3. METHOD 

A single, overarching research question guided 
the design of this study: what is the potential 
difference in learning experience in a web 
development course between undergraduate 
students with access to cloud-based services 
compared to undergraduate students with access 
to local computing services only ? Operationally, 
learning experience appeared too abstract and 
did not adequately establish a measurable critical 
success factor. Based on research by (Schiefele, 
1991), more measurable elements such as time, 
effort, errors, experience, and grades better 
served as critical success factors indicative of 
learning experience. Accordingly, we 
operationalized the general research question 
into five specific sub-questions (Table 1). Further, 
the five sub-questions then informed both the 
identification of the study variables as well as the 
selection of relevant questions for the data 
collection instrument. 

Population 

The population considered in this research was 
limited to four-year college students in the United 
States enrolled in a computer information 
systems or equivalent degree track. We assumed 
that members of the population engage in at least 
one quarter or semester of web development 
study. This assumption appeared rationale upon 
review of existing computer information systems 
and equivalent curricula. Further, we assumed 
that web development courses involved non¬ 
trivial coding work using Microsoft technologies 
(e.g., ASP.NET, C#). Based on the existing 
curricula surveyed, such an assumption appeared 
to be rational. 

Sample 

A self-selection sampling technique, using two 
steps, was employed. To avoid potential sampling 
bias, one of us not involved in teaching the web 
development course issued a call for 
participation. Students responding to the 
solicitation were permitted to self-select into one 


of two study groups (cloud-based versus local- 
based). We did not purposefully filter participants 
based on demographics. However, diversity 
across the institution and within the degree track 
under study facilitated a student population 
representative of the general population. 

Table 1 

List of Operational Research Questions 
Index _ Questions _ 

What is the potential difference in the 
time spent installing and configuring 
the development environment 

1 between students using a cloud-based 
environment compared to students 
using a local environment? 

What is the potential difference in the 
effort spent installing and configuring 
the development environment 

2 between students using a cloud-based 
environment compared to students 
using a local environment? 

What is the potential difference in the 
number of errors arising during 
installation, configuring, and usage of 

3 the development environment 
between students using a cloud-based 
environment compared to students 
using a local environment? 

What is the potential difference in 
normative experience installing, 
configuring, and using the 

4 development environment between 
students using a cloud-based 
environment compared to students 
using a local environment? 

What is the potential difference in the 
final course grades earned by 
i- students using a cloud-based 

development environment compared 
to students using a local development 
environment? 


Two factors bound the call for participation: (a) 
the scope as defined by the purpose of the 
research; (b) volunteers possessing 
characteristics of the broader population. Self¬ 
selection sampling was appropriate to segment 
potential volunteers from the researchers 
(Rutherford, 2006; Salkind & Rainwater, 2003). 
Segmentation (or blinding) was desirable to 
reduce participant selection time and to ensure 
adequate dedication to the study once enrolled 
(Salkind & Rainwater, 2003). Further, despite not 
having the power of probabilistic sampling 
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techniques, self-selected sampling is effective 
when employing experimental research designs 
(Rutherford, 2006). 

Twenty-nine students (out of 65 total) 
volunteered from two sections of an 
undergraduate web development course. The 
course is a core requirement for students in the 
computer information systems degree program 
and may serve as an elective for computer 
science majors. The course required ten weekly 
development projects, a midterm, and a final 
project. The weekly assignments were of 
increasing difficulty (ranging from simple HTML to 
single-page web applications) and included a 
web-based version of the classic game, hangman, 
using MicrosoftASP.NET with C#. 

Fourteen participants joined the cloud-based 
study group with the remaining 15 selecting the 
local-based study group. Participants remained in 
the self-selected study groups for the duration of 
the course. The sample was independent because 
data from one group could not influence data 
from the other (Huck, 2012). Lastly, the sample 
size was appropriate given the scope and design 
of the research (Creswell, 2012; Huck 2012; 
Salkind & Rainwater, 2003). 


Table 2 

Breakdown of variables 


Variable 

Index 

Variable Name 

Variable Type 

10 

Cloud-based 

Independent- 

Categorical 

11 

Local-based 

Independent- 

Categorical 

DO 

Time 

Dependent- 

Ordinal 

D1 

Effort 

Dependent- 

Ordinal 

D2 

Errors 

Dependent- 

Ordinal 

D3 

Experience 

Dependent- 

Ordinal 

D4 

Grade 

Dependent- 

Ordinal 


Study design 

Analysis of the research questions indicated that 
a quantitative design would be necessary 
(Creswell, 2012). More specifically, because the 
research questions inquired how two independent 
groups might comparatively reveal the effect of 
treatment (Salkind & Rainwater, 2003), we used 
a between-subjects experimental design. 
Between-subjects design facilitates applying a 
treatment (i.e., change in development 


environment) to a study group while applying a 
different treatment to the second study group 
(Bellemare, Bissonnette, & Kroger, 2014). 
Additionally, the research questions outlined the 
study variables. 

Variables 

The study included a single independent variable 
(i.e., development environment) with two levels 
(i.e., cloud-based and local-based), and four 
dependent variables (Table 2). Independent 
variable levels aligned with the study groups and 
were categorical in nature. The five dependent 
variables were present in both study groups and 
typed as ordinal. Dependent variables served as 
data collection elements within the study 
instrument and formed a basis for hypothesis 
testing. There was a consideration for controlling 
potential extraneous factors such as prior 
experience with development tools or retaking 
the course after a failing grade. However, we 
decided that allowing for such factors more 
appropriately represented the general population 
and had equal probability of appearing in both 
groups. 

Data Collection 

Data were collected using a web-based 
questionnaire instrument. The instrument 
consisted of nine bounded questions and a single 
unbounded question designed to collect data 
aligned with the dependent variables. The 
bounded questions aligned with the dependent 
variables (Table 2). Six of the bounded questions 
were Likert items with the same scale. The Likert 
scale ranged from strongly agree, to neutral and 
strongly disagree. Three of the bounded 
questions were multiple-choice. All multiple- 
choice questions were of different response 
scales. The last question was unbounded to allow 
participants to submit unfiltered feedback, but 
data were used only as a measure of course 
feedback, not as data in this study's analyses. 
One of us who led the course sections paired 
study participants with each student's final course 
grade. 

A pilot test validated the data collection 
instrument as reliable and internally valid. The 
pilot test included five participants. Pilot 
participants completed the questionnaire and 
submitted feedback to a brief set of meta¬ 
questions. The meta-questions included elements 
such as length of the study instrument, clarity of 
questions, and wording. Additionally, pilot data 
were used to validate the data analysis process. 
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Data analysis 

The data were ordinal as such represent ordered 
categories (Huck, 2012). Interval and ratio data 
types were ruled out because the quantitative 
values collected were not equidistant (Sullivan & 
Artino, 2013) or inclusive of a true zero value 
(Jamieson, 2004). Moreover, since the Likert 
items and multiple-choice questions would be 
inferentially analyzed, we felt that treating such 
as ordinal would be appropriate (Lovelace & 
Brickman, 2013; Norman, 2010). 

A two-phased data analysis procedure was used 
first to describe the data and then to test a (null) 
hypothesis using an inferential statistic. The 
research hypothesis was that students using a 
cloud-based service to host a web development 
environment would (a) spend less time installing 
and configuring the development environment; 
(b) expend less effort; (c) encounter fewer 
errors; (d) report a higher quality learning 
experience; (e) finish with higher course grades. 
Conversely, the null hypothesis considered in this 
study posited that students using a cloud-based 
service to host a web development environment 
would (a) spend the same or more time installing 
and configuring the development environment; 
(b) expend the same or more effort; (c) 
encounter the same or more errors; (d) report the 
same or a lower quality learning experience; (e) 
finish with equivalent or lower course grades. 
However, like with the research questions, we 
operationalized the general hypotheses into more 
specific, more testable statements (Table 3). 


Table 3. 

Summary of Operationalized Hypotheses 



Research 

Hypothesis 

Element 

Null 

Hypothesis 

Element 

Installing & 

HAi - Less 

HO! - Same or 

configuring 

time 

more time 

Effort 

HA 2 - Less 

H0 2 - Same or 
more 

Errors 

HA 3 - Less 

H0 3 ~ Same or 
more 

Learning 

HA 4 - Higher 

HO 4 - Same or 

experience 

quality 

lower quality 

Grades 

HAs - Higher 

HOs- Same or 
lower 


The Mann-Whitney U statistic was the appropriate 
inferential statistical to test the null hypotheses. 
The rationale for the decision included 
independence of study groups, data type 
(ordinal), and most importantly because the 
research questions required comparison of the 


two study groups (Huck, 2012; Salkind & 
Rainwater, 2003). Alternative statistical 
measures were not appropriate (e.g. Wilcoxon, 
Kruskal-Wallis) because of the number of study 
groups, uneven samples, and data type 
(McDonald, 2014). 

4. RESULTS 

Data were first analyzed using descriptive 
statistical techniques. Doing so permitted a 
general understanding of how participants 
engaged with the associated development 
environments. As well, the descriptive indicators 
facilitated quick, visual interpretation of the data. 
For readability, the descriptive results are 
organized in order of appearance on the data 
collection instrument. 

The first question asked participants to estimate 
the total time spent downloading, installing, and 
configuring the development environment (Figure 
Al). The question was bounded, multiple-choice 
with six response options. Overall, the cloud- 
based group (between 554 and 1140 minutes 
total) described spending less time downloading, 
installing, and configuring the development 
environment compared to the local-based group 
(between 885 and 1260 minutes total). Further, 
the cloud-based group demonstrated a central 
tendency lower ( Mdri 2 = 5 or 31-60 minutes) 
compared to the local-based group (Mdrii = 4 or 
61-120 minutes). Overall, both groups 
demonstrated equal distribution of participants in 
the lowest response range (A/ = 4 at < 30 
minutes) while the local-based group alone 
demonstrated more participants at the highest 
response ranges (A/ = 2 at 121 to 180 minutes; N 
= 1 at > 241 minutes). 

The second question asked participants to gauge 
the level of effort or difficulty expended to 
download, install and configure the development 
environment (Figure A2). The question was 
bounded, multiple-choice, with three response 
options. Overall, the local-based study group 
described (Mdrii = 3 or Easy) the effort as easier 
compared to the cloud-based group ( Mdn 2 = 2 or 
Moderate). Both groups demonstrated equal 
distribution at the most difficult selection option 
(A/ = 1 at Hard). However, the local-based group 
demonstrated a higher number of participants 
describing effort in the easiest selection option (A/ 
= 9 at Easy) compared to the cloud-based group 
(A/ = 4 at Easy). The inverse appeared for the 
middle selection option; local-based participants 
were few (A/ = 5 at Moderate) compared to cloud- 
based participants (A/ = 9 at Moderate). 
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The next question asked participants to select a 
level of agreement with the statement, my 
development environment contributed in a 
positive manner to my web development 
experience (Figure A3). The question was 
bounded and consisted of a five-element Likert 
scale. The two study groups differed in level of 
agreement by one degree according to central 
tendency: local-based participants agreed with 
the statement (Mdni = 4) while the cloud-based 
group distributed between disagreement and 
neutral ( Mdri 2 = 2.5). Collectively, more local- 
based participants gravitated towards agreement 
(A/ = 5 at Strongly Agree; N = 3 at Agree) while 
cloud-based participants grouped towards 
disagreement (A/ = 1 at Strongly Disagree; N = 2 
at Disagree). 

Next, participants offered a level of agreement 
with the statement; I was able to complete my 
assignments without error because of my 
development environment (Figure A4). The 
question was bounded and used a five-element 
Likert scale. Overall, local-based participants 
agreed with the statement ( Mdni= 4) while the 
cloud-based group remained neutral ( Mdn 2 = 3) 
again. Further, the two groups' responses were 
distributed similarly to the prior question insofar 
as the majority of local-based participants 
selected levels of agreement (A/ = 3 at Strongly 
Agree; N = 8 at Agree) compared to the cloud- 
based group converging towards levels of 
disagreement (A/ = 1 at Strongly Disagree; N = 4 
at Disagree). 

The fifth question was bounded and used a five- 
element Likert scale to collect participant level of 
agreement with the statement, my environment 
allowed for the efficient installation of 
development tools (Figure A5). The question was 
bounded within a five-element Likert scale. The 
local-based development environment study 
group predominantly agreed ( Mdni= 4) while the 
cloud-based study group centrally distributed 
between neutral and agreement ( Mdn 2 = 3.5). 
Both groups gravitated towards aggregate levels 
of agreement. However, cloud-based participants 
were more numerous across disagreement 
elements (A/ = 3 at Disagree; N = 1 at Strongly 
Disagree) compared to the local-based group (A/ 
= 1 at Disagree). 

Participants selected a level of agreement with 
the development environment facilitating the 
efficient completion of coursework in the sixth 
question (Figure A6). The question used a five- 
element Likert scale and was bounded. 
Participants in the local-based development 
environment group generally agreed ( Mdni = 4). 


Moreover, more than one-third of individuals in 
the local-based study group (A/ = 6) described 
strong agreement with the statement. 
Comparatively, cloud-based participants were 
overall neutral ( Mdn 2 = 3) but with equal 
distribution of individuals describing strong 
agreement (A/ = 4) as well as disagreement (A/ = 
4). 

The next question asked participants to select a 
level of agreement with the statement; I was able 
to rapidly build my web development 
environment (Figure A7). The question consisted 
of five agreement level options within a bounded 
Likert scale. Overall, the two study groups 
described identical levels of strong agreement (A/ 
= 4, each) and levels of agreement differing by 
one degree (Mdni = 4; Mdn 2 = 3). As well, strong 
disagreement levels were identical between 
groups (A/ = 1, each). On the other hand, a 
proportionally large number of participants in the 
local-based group selected a neutral level of 
agreement (A/ = 6) while few (A/ = 2) cloud-based 
participants did so. 

The eighth question posed to participants inquired 
if learning was effective and efficient because of 
the associated development environment (Figure 
A8). The question was bounded and measured 
agreement across a five-element Likert scale. 
Members of the local-based study group 
centralized upon agreement (Mdni = 4) whereas 
cloud-based participants tended to describe a 
neutral position ( Mdn 2 = 3). Additionally, there 
were zero cloud-based participants present in the 
Likert scalar extremes. Conversely, the local- 
based study group was heavily represented 
across both agreement elements (A/ = 5 at 
Strongly Agree; N = 6 at Agree). 

The last instrument question posed to study 
participants inquired about the number of errors 
encountered during the download, installation, 
and configuration of the development 
environment (Figure A9). The question was 
bounded and used a five-element multiple-choice 
scale. Overall, both groups described 
encountering errors. Local-based participants 
indicated a lower error incidence (Mdni = 5 at 
between zero and one errors) compared to the 
cloud-based study group (Mdn 2 = 4 at between 
one and two errors). Furthermore, participants 
using a local-based development environment 
described encountering between 9 and 24 errors 
in total. In contrast, cloud-based participants 
described encountering between 20 and 34 total 
errors. 
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Participants across both study groups performed 
well with respect to final course grades (Figure 
A10). Participants in the local-based development 
environment group demonstrated a higher grade 
tendency ( Mdrh = 4 or A) than the cloud-based 
group ( Mdn 2 = 3.7 or A-). Furthermore, both 
groups aligned the tendency of the general 
student sample frame in the course sections used 
in this study. 

Inferential Analysis 

Inferential data analysis was used to measure the 
differences between the two study groups 
according to the research questions and 
hypotheses. The inferential test results are 
collated below according to such groupings. 
Furthermore, independent variables of cloud- 
based development environment group (i.e., 
Azure) and local-based development environment 
group were coded as CBDE and LBDE for 
readability. 

Installing and configuration 

We first analyzed data from two questions as a 
comparative measure the HOi hypothesis. 
Descriptive testing of question one indicated that 
the time spent installing and configuring the 
development environment was lower for the 
CBDE group (31-60 minutes) than for the LBDE 
group (61-120 minutes). However, the Mann- 
Whitney U was found to be 90.5 (p > 0.05) and 
so the null hypothesis could not be rejected 
(Table 4). Further, the results were not 
statistically significant 


Table 4 

Time downloading , installing , and configuring 
t he development environment between groups 


Ranks 

Group 

N 

Mean 

Rank 

Sum of Ranks 

CBDE 

14 

16.04 

224.5 

LBDE 

15 

14.03 

210.5 

Total 

29 



Mann-Whitney U test results 

U 

Z 

P~ 

value 

SD 

90.5 

- 

0.542 

22.91 


0.611 



Notes: Ql; 
.80 

p = 0.05 one-tailed; power at > 


Concurrently, we analyzed question five. 
Descriptive testing indicated that the LBDE and 
CBDE groups were close in estimating efficiency 
in installing development tools within associated 
environment. The Mann-Whitney U was 66 (p > 
0.05) so the null hypothesis could not be rejected 


(Table 5). The results were not statistically 
significant. 


Table 5 

Efficient installation of development tools 
between groups _ 


Ranks 

Group 

N 

Mean 

Rank 

Sum of Ranks 

CBDE 

14 

12.21 

171 

LBDE 

15 

17.6 

264 

Total 

29 



Mann-Whitney U test results 

U 

Z 

P~ 

value 

SD 

66 

1.68 

0.093 

22.91 

Notes: Q5, 
.80 

’ p = 0.05 one-tailed; power at > 


Effort 

Secondly, we analyzed data from two additional 
questions as a comparative measure of the HO 2 
hypothesis. Descriptive analysis of question two 
indicated that the effort expended to install and 
configure the development was greater for the 
CBDE group (Easy) than for the LBDE (Moderate). 
The Mann-Whitney U was 74 (p > 0.05) so the 
null hypothesis could not be rejected (Table 6). 
Moreover, the results of the test were not 
statistically significant. 


Table 6 

Effort expended to install and configure 
development environments between groups 


Ranks 

Group 

N 

Mean 

Rank 

Sum of Ranks 

CBDE 

14 

12.79 

179 

LBDE 

15 

17.07 

256 

Total 

29 



Mann-Whitney U test results 

U 

Z 

P~ 

value 

SD 

74 

1.331 

0.184 

22.91 

Notes: Q2; 
.80 

' p = 0.05 one-tailed; power at > 


In tandem, descriptive analysis of data from 
question seven, revealed similar information as 
question two. What is more, the Mann-Whitney U 
was 105 (p > 0.05). Such indicated that the null 
hypothesis, again, could not be rejected (Table 
7). The results were not statistically significant 
though. 
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Table 7 

Rapidity of building the development 
environment between groups _ 


Ranks 

Group 

N 

Mean 

Rank 

Sum of Ranks 

CBDE 

14 

15 

210 

LBDE 

15 

15 

225 

Total 

29 



Mann-Whitney U test results 

U 

Z 

P~ 

value 

SD 

105 

0.022 

0.984 

22.91 

Notes: Q7; p = 0.05 one-tailed; power at > 

.80 


Errors 

Next, we analyzed data from questions four and 
nine as a comparative measure of the HO 3 
hypothesis. Question four revealed that the LBDE 
group agreed that group participants were able to 
complete assignments without error due to the 
local-based development environment while the 
CBDE group remained neutrally aligned. The 
Mann-Whitney U was 59 (p < 0.05) and would 
support rejection of the null hypothesis. The 
results (Table 8) were statistically significant. 


Table 8 

Completion of assignments without errors 
between groups _ 


Ranks 

Group 

N 

Mean 

Rank 

Sum of Ranks 

CBDE 

14 

11.71 

164 

LBDE 

15 

18.07 

271 

Total 

29 



Mann-Whitney U test results 

U 

Z 

P~ 

value 

SD 

59 

1.99 

0.047 

22.91 

Notes: Q4; 
.80 

’ p = 0.05 one-tailed; power at > 


Analysis of question nine demonstrated a 
competing view of the comparison between study 
groups however. Descriptively, the LBDE study 
group reported fewer errors compared to the 
CBDE group. Yet, the Mann-Whitney U was 76.5 
(p > 0.05) and would not support rejection of the 
null hypothesis (Table 9). Moreover, the results 
were statistically significant. 


Table 9 

Estimated number of errors during installation , 
configuration and use of development 
environments between groups _ 


Ranks 

Group 

N 

Mean 

Rank 

Sum of Ranks 

CBDE 

14 

12.96 

181.5 

LBDE 

15 

16.9 

253.5 

Total 

29 



Mann-Whitney U test results 

U 

Z 

P~ 

value 

SD 

76.5 

1.222 

0.111 

22.91 

Notes: Q9 t 
.80 

; p = 0.05 one-tailed; power at > 


Learning experience 

Penultimate hypothesis (F/0 4 ) testing drew upon 
data from three questions. Question three 
descriptively indicated the LBDE group agreed 
that the environment contributed in a positive 
manner to the learning experience while CBDE 
participants vacillated between disagreement and 
neutrality. The Mann-Whitney U was 59 (p < 
0.05) and data suggested that the null hypothesis 
could be rejected (Table 10). Furthermore, the 
results were statistically significant. 


Table 10 

Contribution of development environment to 
positive learning experience between groups 


Ranks 

Group 

N 

Mean 

Rank 

Sum of Ranks 

CBDE 

14 

18.07 

271 

LBDE 

15 

11.71 

164 

Total 

29 



Mann-Whitney U test results 

U 

Z 

P- 

value 

SD 

59 

1.99 

0.047 

22.91 

Notes: Q3; 
.80 

' p = 0.05 one-tailed; power at > 


Next, we analyzed data from question six. 
Descriptive analysis demonstrated that the LBDE 
agreed that the development environment 
facilitated the efficient completion of coursework 
while the CBDE group trended between neutral 
and agreement. The Mann-Whitney U was 73.5 
(p > 0.05). The null hypothesis could not be 
rejected based on these data. As well, the results 
(Table 11) were not statistically significant. 
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Table 11 

Facilitation of coursework completion within 
development environments between groups 


Ranks 

Group 

N 

Mean 

Rank 

Sum of Ranks 

CBDE 

14 

12.75 

178.5 

LBDE 

15 

17.1 

256.5 

Total 

29 



Mann-Whitney U test results 

U 

Z 

P~ 

value 

SD 

73.5 

1.35 

0.089 

22.91 

Notes: Q6; 

p = 0.05 one-tailed; power at > 

.80 





Lastly, descriptive testing of question eight found 
LBDE group members agreed that the learning 
experience was efficient and effective within the 
development environment whereas the CBDE was 
neutral. The Mann-Whitney U was 105 (p > 0.05). 
The null hypothesis could not be rejected. The 
results of the test were not statistically significant 
(Table 12). 


Table 12 

Efficiency and effectiveness of learning 
experience between groups _ 


Ranks 

Group 

N 

Mean 

Rank 

Sum of Ranks 

CBDE 

14 

15 

210 

LBDE 

15 

15 

225 

Total 

29 



Mann-Whitney U test results 

U 

Z 

P- 

value 

SD 

105 

0.022 

0.984 

22.91 

Notes: Q8, 
.80 

’ p = 0.05 one-tailed; power at > 


Grades 

Final grades were collected directly from the 
course learning management system portals. 
Descriptively, data revealed that the LBDE group 
earned higher marks (A or 4.0) than for the CBDE 
group (A- or 3.7). The Mann-Whitney U was 53 (p 
< 0.05) and thus the null hypothesis could be 
rejected (Table 13). As well, the results were 
statistically significant. 


Table 13 


Earned grades between groups 


Ranks 

Group 

N 

Mean 

Rank 

Sum of Ranks 

CBDE 

14 

11.29 

158 

LBDE 

15 

18.47 

277 

Total 

29 



Mann-Whitney U test results 

U 

Z 

P~ 

value 

SD 

53 

2.247 

0.024 

22.91 

Notes: p = 

0.05 one-tailed; 

power at > .80 


5. CONCLUSIONS 


This project was limited by the fact that students 
were able to choose between stand-alone 
computers and a cloud solution. As a result, the 
cloud solution did not take advantage of 
connecting to other Internet resources. Students 
completed the same web project whether or not 
they used the cloud or a personal computer. 
Future studies need to explore a rich connectivity 
environment in which students connect their 
website to other systems via the Internet. 

Furthermore, student performance with the CBDE 
environment (Azure) demonstrated that students 
encountered greater difficulties than anticipated 
in gaining proficiency with the technology. If 
students had been given an introductory 
exposure to Azure, the outcome of this study may 
have changed dramatically. Just the fact that 
students knew they were in a metered 
environment seemed to cause students to be less 
playful with the system, which perhaps hindered 
students' learning processes. It will be important 
to determine whether the introductory exposure 
to the cloud should be in a prerequisite course, or 
an introduction to cloud computing within the web 
course. 

Installation and configuration 

The inability to reject the null hypothesis and the 
lack of statistical significance between the CBDE 
and LBDE participants revealed in question 1 was 
a surprise. Likewise, the greater efficiency of 
working on local machines was not expected. 

Students in the class have been working with 
personal computers for years, however, for most 
in the CBDE group, this was their first experience 
with a cloud computing environment. Students' 
lack of familiarity with the environment may have 
led to inefficiencies. 
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It is reasonable to surmise that students with less 
capable personal computers, opted for the cloud 
computing environment and may, in fact, have 
had a better experience than would have been the 
case using their personal computers. 

A discussion amongst faculty who teach the web 
course reveals that there are often between three 
and eight students who dedicate more than 240 
minutes to installing and configuring their 
development environments in a given section of 
this class. So the outcome in this study is viewed 
as positive despite the inability of the results to 
discern variance between the CBDE and LBDE 
installation groups. 

Future studies may need to include sections of the 
course in which students do not have the cloud 
option available and sections where the cloud 
service is the only option available to tease out 
the impact on students' time. Alternatively, 
perhaps a within-subjects design in which each 
student completes both scenarios will more fully 
assess the impact of differences in time 
consumption. 

Effort 

Question 2 reveals that the LBDE group met with 
less difficulty downloading, installing and 
configuring the development environment. The 
agreement in question 7 demonstrating that 
students in the LBDE group built their 
environments more rapidly was therefore not a 
surprise. Students familiarity with personal 
computers allows them to work effectively in this 
environment. Also, students in the CBDE group 
experienced connectivity difficulties at several 
points during the academic term that grew 
increasingly exasperating as deadlines drew near. 
Future studies will need to measure or control for 
connectivity issues to assess their impact. 

This outcome raises the question if experience 
and time spent in cloud computing environments 
will allow students to gain a measure of 
effectiveness using the cloud that rivals current 
personal computer use. More effective training 
for students on the utilization of the cloud, 
coupled with increased time on tasks in the cloud, 
appear to be needed. 

Errors 

Questions 4 and 9 reveal that students in the 
LBDE group perceived they were better able to 
complete assignments without error and also 
reported fewer errors. This outcome was a 
surprise as the cloud operator specifically 
supported the IDE in use, and the environment 
was well suited to the IDE's installation. 


One plausible explanation for the discrepancy lies 
with the connectivity issues that students 
experienced in the cloud group. Students' lack of 
familiarity with the system, coupled with erratic 
connectivity, may have caused students to 
believe their input did not register and the 
command to be repeated with undesirable 
outcomes. 

Learning Experience 

Question 3 results were significant and reveal that 
the LBDE group believed their learning 
environment better contributed to their learning 
experience. The results of questions 6 and 8, 
while not statistically significant, suggest the 
LBDE group also viewed their environment as 
better supporting the efficient completion of 
assignments and more effective and efficient 
learning experiences. 

Learning experience results are perhaps the most 
troubling. Student confidence in the efficacy of 
their learning environment is of the utmost 
importance. A positive learning experience is 
critical to enhancing the utility of the cloud 
computing option to gain broad acceptance from 
students. 

Grades 

While the LBDE group earned significantly higher 
grades, it is also true that the class overall scored 
higher than normal and the distribution of scores 
was more consistent than typical sections of the 
course. The cloud environment is believed to be 
at least partly responsible for the higher and more 
consistent grades as students who may otherwise 
have struggled with an ill-equipped computer 
used the cloud instead. There also seemed to be 
a novelty factor that enamored students and may 
have led to increased time on task and therefore 
better grades. 
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Figure 1. Minutes spent downloading, installing, and configuring development environments. 



15 (4) 
July 2017 


Figure 2. The perceived level of effort or difficulty downloading, installing, and configuring the 
development environment. 
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Figure 3. Participant level of agreement with the notion that the development environment contributed 
positively to the learning experience. 



SA A N D 0 SD 

Level of Agreement 


Figure 4. Participant agreement levels associated with the ability to complete assignments without 
error due to the development environment. 


©2017 ISCAP (Information Systems and Computing Academic Professionals) 
http://www.isedj.org; http://iscap.info 


Page 65 




















































































Information Systems Education Journal (ISEDJ) 
ISSN: 1545-679X 


15 (4) 
July 2017 



Figure 5. Participant level of agreement with level of efficiency in installing development tools within 
the associated group environment. 
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Figure 6. Distribution of participant levels of agreement with the development environment facilitating 
efficient completion of assignments. 
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Figure 7. Participant levels of agreement with the notion that the associated development environment 
was built with rapidity. 
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Figure 8. Participants' level of agreement relative to the development environment leading towards 
effective and efficient learning experiences. 
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Number of Errors. 

Figure 9. The number of errors encountered by participants during downloading, installation, and 
configuration of the development environment. 



Figure 10. Distribution of final grades earned by participants using local-based development 
environments compared to cloud-based development environments. 
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